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This instructional development project evaluates the effect of a matrix-based strategy to assist multimedia 
authors in acquiring and applying principles for effective multimedia design. The Multimedia Matrix , based on the 
Park and Hannafin (1993) Twenty Principles and Implications for Interactive Multimedia design , displays a 
condensed version of those Principles . Space is provided for students to generate their own responses to the presence 
or absence of elements as they evaluate multimedia modules before and after they begin their own designs. 

Instruction followed Jacobson's (1994) Theory-to-Design framework. Formative evaluation of the project is based 
on qualitative and quantitative data reflecting perceptions of graduate students ; enrolled in a course containing a 
multimedia authoring component , toward this Matrix-based approach. Their perceptions toward the Matrix-based 
authoring instructional design and their and the evaluators' analyses of the students' subsequent multimedia 
productions are presented and analyzed in relation to the effectiveness of applying two versions of the Multimedia 
Matrix. Implications for future application and investigation of the Matrix is provided. 



The increase in authoring environments for multimedia 3 design has not necessarily lead to increased quality 
in the resulting productions (Locatis, Ullmer, Car, Bernard, Le, Lo, & Williamson, 1992; Jones & Smith, 1992; 
Stanton & Baber, 1994); yet scant guidance is offered to designers to connect them with relevant research and 
learning theory (Ambrose, 1991). To provide guidance for multimedia designers concerned with developing quality 
modules, Park and Hannafin (1993), developed Principles and Implications for Interactive Multimedia Design based 
on twenty psychological, pedagogical, and technological foundations from research findings. For example, to apply 
their Principle 15 — “Learners become confused and disoriented when procedures are complex, insufficient, or 
inconsistent.” — they suggest that the designer should, “Provide clearly defined procedures for navigating within the 
system and accessing on-line support.” (Park & Hannafin, 199 p. 77-78). 

As novice and intermediate level designers attempt to incorporate these Princip les into their multimedia 
productions, while simultaneously integrating technical knowledge of authoring programs and subject matter 
knowledge, they face challenging, higher order cognitive demands. In such situations, research in cognitive science 
and instructional technology, suggests that integrated, contextually-based instructional strategies be considered 
(Tennyson, Elmore, & Snyder, 1992). 

Many beginning hypermedia authors tend to be teachers who have subject matter expertise and little or no 
knowledge about technical aspects of multimedia authoring. No specific guides exist for introducing authoring 
concepts such as those encompassed by the Principles . An attempt prior to this investigation, to introduce the 
Principles using a broad-based approach of expository techniques of assigned reading and discussion of the Park and 
Hannafin article, a demonstration, a brief Principles checklist, and encouragement of incorporation of relevant 
principles into initial multimedia productions, resulted in modules that were judged by the both the students and the 
instructor to contain only some aspects of the Principles . The majority of the students, however, expressed concern 
over the challenges posed by trying to remember the Principles while working in the multimedia environments. 
(Sherry, 1995). Discussion with one of the authors of the Principles (R. Hannafin, personal communication, 
February 16, 1995), resulted in agreement about the potential usefulness of developing a visual guide, hereafter 
referred to as the Matrix. An instructional design was developed that included the use of a Matrix crafted to serve as a 
visual, cognitive reference with which students could generate specific responses as they worked in a contextually 
relevant setting. 

It is not surprising that research on the effectiveness of matrices to concepts, is mixed (Newell & Olejnik, 
1982; Rumelhart, 1980; Siebold, 1989; Spiegel & Barufaldi, 1994), given the great diversity of matrix designs and 
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3 Multimedia — "the use of multiple formats for the presentation of information, including text, 
still or animated graphics, movie segments, video, and audio information. Computer-based interactive 
multimedia includes hypermedia and hypertext." (Tolhurst, 1995, p. 25). permission to reproduce this 
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usage reported in such studies. By employing a systematized approach to usage, combined with feedback from the 
participants, a critical element in effective evaluation (Reiser & Kegelmann, 1994), a Matrix-based instructional 
design was developed and refined through the testing of two versions of the Matrix developed to specifically guide 
users in applying the Principles . 

Method 

Participants 

Seventeen graduate level education majors, involved in multimedia development participated in providing 
preliminary data on the Matrix-based instructional design project. Technical abilities ranged from beginner to 
advanced with all having no prior knowledge of the Park and Hannafin (1993) Principles, 

Procedures 

Instruction was based on the seven steps of Jacobson* s Theory-to-Design framework (1994), that employs 
cognitive flexibility theory (CFT) to support the “development of flexible representation of knowledge that will help 
promote deep conceptual understanding and ability to adaptively use knowledge in new situations” 

(p. 146). The seven steps that were followed were: 

1) Employing rich cases and examples — An exact statement of the twenty Principles and an abbreviated 
version of implications for applying the Principles was developed and displayed on versions one and two of a Matrix 
created for student use. Blank cells were provided for making notations about key elements of effective multimedia 
authoring that could be identified during a review of multimedia modules. The Park and Hannafin article and five 
examples of student-produced multimedia modules were provided along with the first version of the Matrix during 
instructor demonstration and student analyses . 

2) Using multiple forms of knowledge representation — The articles, the Matrix v.I, the modules, and a 
talk-aloud approach were used to present the information in multiple forms. Students, working in self-selected 
dyads, chose three of the five multimedia modules to evaluate using the first Matrix as a guide. 

3) Linking abstract concepts to case examples — A variety of the abstract Principles were contained in the 
sample modules. 

4) Demonstrating conceptual complexities and irregularities — The modules that served as examples varied 
in the quality and quantity of complex features of multimedia Principles that they incorporated. 

5) Stressing the interrelated and weblike nature of knowledge — The Matrix was designed to be interrelated 
to the Park and Hannafin article. Version one was used when the students evaluated multimedia modules developed 
by others; version two was used when they evaluated their own modules. 

6) Encouraging knowledge assembly from different conceptual case sources — The students were required to 
analyze modules in writing, as well as to construct their own analysis of the effectiveness of the first Matrix. Their 
suggestions for change were incorporated in the second version of the Matrix. 

7) Promoting active learning — Throughout their authoring experience, students were actively engaged with 
analyzing and applying both versions of the Matrix, the article, and the modules and with constructing their versions 
of the Matrix approach. 

Data Analysis 

Quantitative and Qualitative Data 

Both quantitative and qualitative data were gathered to evaluate the overall effectiveness of the Matrix-based 
instructional design and the students’ ability to apply the Principles to their designs. Quantitative data consist of 
students’ mean scores on aspects of the usefulness of the multimedia production and of the Principles , as well as on 
the overall instructional design of the course. Quantitative data consisted of interval data from a written survey 
administered at the end of the course; the percentage of completed multimedia productions judged by the student 
authors as being worthwhile additions to the software preview center of the college; the degree of effectiveness that 
the students reflected toward the two versions of the Matrix as rated by two evaluators; and the percentage of students 
meeting criterion-based measures for presentations and reports that reflected attainment of declarative and procedural 
outcomes as judged by instructor ratings. Qualitative data are reported in an analysis and ranking of comments from 
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the students about both the Matrix and their modules in relation to application of principles for effective multimedia 
design and in the instructor’s overall evaluation of all components of the evaluation process. 

Formative Evaluation — Matrix 

Phase I. In Phase I students were asked to provide written evaluations of the first version of the Matrix in 
terms of its positive and negative effects. Their suggestions for changes to the guide were also sought. Two 
evaluators analyzed these responses to identify commonalties in the comments* The resulting themes were ranked 
according to the most frequently occurring items. Based on this analysis, changes were made to the first version of 
the Matrix. The revised Matrix v. 2, was made available to all the students during their subsequent multimedia 
authoring. 

Phase II. In Phase II, the students’ multimedia productions were evaluated by the instructor in relation to 
their ability to reflect the Principles in their productions. A minimum of ten of the twenty principles was set as the 
baseline for effective design for their productions. An option was provided for students to analyze their modules in 
writing in relation to Matrix v.2 and to provide a log of the time spent using the matrices. 

Students provided additional written responses in relation to the interactive multimedia component of the 
course by responding to three items on a Likert-like survey administered at the conclusion of the course. 

Results and Discussion 

This evaluation emphasized listening to student opinions, as this approach was determined to have the 
greatest potential for providing relevant data on ways to improve discrete elements of the multimedia module, as well 
as the overall design of the course (English & Reigeluth, 1996; Lohr, Ross, & Morrison, 1995). The Matrix-based 
course component was perceived by learners to effectively contribute to their intellectual curiosity (4.5), their 
satisfaction (4.5), and general perceived usefulness (4.5) as indicated by learner responses to a survey administered at 
the end of the course, that contained a Likert-like scale for responses that ranged from 1 (almost never or almost 
nothing) to 5 (almost always, a great deal). Of all the course components, multimedia production was perceived to 
be the most useful one. 

In analyzing the students’ perceptions about both versions of the Matrix, comments offered by more than 
25% of the students were reviewed. As shown in Table 1, within those parameters, the students provided more 
negative and change comments than positive ones for the first version of the Matrix. 

Based on these evaluations, changes were made to the text of the Matrix to clarify vocabulary and redundant 
information. A checklist format was adopted for greater consistency and ease of use. The response boxes underwent 
minor reformatting to systematize the overall design of the instrument. To clearly keep the relationship of the 
Matrix to its accompanying article, no additional items were added, although some students had made that request. 
The revised Matrix is depicted in Figure 1. 

The analysis of the revised Matrix, indicates a major change with comments in the positive category 
outnumbering the negative and change categories. According to students the new version does achieve its purpose of 
simplifying the application of the Principles for the authors. This comment was provided by 82% of the respondents 
and achieved the top ranking for all items. 

While the main negative comments that appeared for the first version did not occur for the second one, the 
shape of the response boxes was still seen as a problem. Although the newer version appears to support users’ needs 
to a greater extent than the initial one, these results should be interpreted with caution. The results of the student- 
based evaluations of Matrix v.2 are based on responses from eleven of the seventeen students. In keeping with the 
constructivist-based design of this part of the course, students could determine whether or not to use and evaluate 
Matrix v.2. Of the six students who did not elect this option, five had had negative reactions toward the first version. 
The effect of these students’ non participation in the final evaluation needs to be kept in mind to avoid interpreting 
the students’ second round of comments in an overly optimistic manner. 

When the students’ views of the Matrices are investigated in relation to their skills as authors — novices or 
more advanced designers — there is a similar pattern within the number of the types of responses each group of 
students offers in relation to positive, negative and change aspects. For example, novices provided more positive 
comments than advanced students and less when making negative responses and suggestions for change. It may be 
that the novices welcomed any type of support, while the advanced students were more discerning about the type of 
support they required. Novices were more apt to mention version one of the Matrix as contributing to guiding 
evaluators and supporting consistent feedback. The advanced students also recognized the contribution of guiding 
evaluators, but were more apt to mention support of consistent feedback and focusing attention. It could be that the 
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novices perceive consistency as requiring less mental effort when considering the Principle s, allowing them to 
concentrate on mastering the authoring program itself, an issue of less concern for the more advanced learners. 

By version two, as the novices had become more experienced in manipulating the authoring program, they 
cited simplifying the application of the Princip les and/or convenience and an effective checklist format as their two 
most frequently mentioned items. The predominant response of the advanced students was the same for both items, 
but they also recognized the Matrix as a guide for designers by mentioning that aspect as frequently as they 
mentioned its checklist feature. As advanced students reported spending less time using the matrices (an average of 
one hour less with Matrix v.2 than the novices), the former students may have found more time for reflecting on the 
effect of the Matrix and, thus, come to this conclusion. 

Regarding negative comments and suggestions for change, fifty percent of the novices stated that Matrix v.l 
was inconsistent if used by many and had vocabulary problems. Only thirty-six percent of the advanced students 
recognized the issue of inconsistency, and as such, it appeared as the second most frequently appearing item for them 
in their negative comments. Their most frequently mentioned negative item was redundancy. Forty-five percent cited 
that issue. After using the second version, the novices showed a marked decline in their negative comments. None 
mentioned inconsistency as a problem and only twenty-five percent continued to cite the vocabulary problem. Issues 
of redundancy and requiring prior knowledge appeared at this stage for twenty-five percent of the novices. It may be 
that as the students became more confident with their developing authoring skills they became more positive. A 
similar change appeared with the advanced students as negative comments declined to only one. That one comment 
regarded the design of the response boxes with forty-three percent criticizing the shape. The appearance of that type of 
comment by a large percentage of the advanced students may be a result of their desire to have more space for their 
comments as these students were observed to be more apt to write remarks than were the novices. 

The change comments followed the pattern for the negative comments. Suggestions for change decreased 
between the two versions of the Matrix. The novices had requested that the initial matrix be reformatted to a 
checklist format and that a provision be added for overall scoring. As both these features were incorporated into the 
revised version, no comments were offered about these topics for Matrix v.2. The advanced students suggested that 
items other than those covered by the twenty Principles be added to the Matrix after they worked with Matrix v.l. 
This request indicated their broader knowledge of other aspects of multimedia that were not addressed by Park and 
Hannafin. As the intent was to create an instructional tool that would support and enhance the Park and Hannafin 
article, no attempt was made to add items to Matrix v.2. This decision was explained to the students. Both the 
novices and the advanced students did request that Matrix v.2 be simplified and shortened to be less time consuming. 
Novices did report spending approximately one hour longer than the advanced students when working with the 
Matrix, with times reported as 2 1/2 hours for the former and 1 1/2 hours for the latter. Such amounts of time, do 
indicate a need to refine the instrument for greater efficiency. 

The results of the quantitative data in the end of the course survey have greater potential for reliably 
reporting the students* view toward the overall instructional aspects of the multimedia component as all students 
provided responses. On a Likert-like scale that ranged from 1 (almost never or almost nothing) to 5 (almost always, 
a great deal), the learners indicated that the multimedia authoring component effectively contributed to their 
intellectual curiosity, their satisfaction, and general usefulness as indicated by a mean of 4.5 for each of the threee 
items. 

Ten modules were developed by the students, five by individuals and five by groups. All modules were 
judged by the instructor to address a minimum of ten of the principles. While this evaluation has the potential for 
instructor bias, the value of the modules was supported by the multimedia authors. Sixteen of the seventeen 
students judged their productions as worthwhile contributions to the Technology Learning Center of the college, a 
site that serves as a preview center for commercially- and student-produced educational software. Peer review and exit 
interviews have been employed with similar findings by Spector, Muraida and Marlino in their study on CBI 
authoring (1992). 

Conclusions and Implications 

Focusing students’ attention on specific content structure, in this case, on critical principles for multimedia 
authoring, has shown indications of increasing structural knowledge and retention (Beasley & Waugh, 1996) and 
keeping students on task (Shore, Erickson, Garick, Hickman, Stanley, Taylor, & Trunfio, 1992). For the demanding 
task of authoring, the Matrix that was offered to support learning, has the potential to decrease demands made on the 
mental effort that the student must expend on remembering authoring Principles and to increase effort toward 
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authoring content. As one student commented, the learners were required to, “really think about the program 
[multimedia module] in terms of how fundamentally it teaches. It gets the evaluator past the ‘bells and whistles’”. A 
balance does need to be struck between modifying the Multimedia Matrix to a point where it becomes less time 
consuming for students to use, yet still retains its connection to the original Park and Hannafin article (1993), as 
well as, continues to encourage students to revise and personalize it. 

While the matrix-based approach to instruction appears to have contributed to the development of the 
multimedia authoring skills for these learners, future studies may reveal that the flexible approach to instruction, 
particularly instruction that supports the learners’ evaluation and adaptation of the Matrix to meet their needs has a 
more significant impact on learning, more than the actual implementation of the Matrix itself (Jacobson, 1994). 
Future investigators may wish to compare the effect of other authoring frameworks, such as the Hypermedia Design 
Model (Garzotto, Mainetti, Paolini, 1995) to the Matrix when studying the implementation of authoring in courses. 
Similar to Jacobson’s Theory-to-Design framework, the matrix-based model appears, in a preliminary analysis, to 
meet some of the design criteria as a guide on discrete aspects of effective multimedia design. 
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Table l . Most Frequently Occurring Comments Concerning Multimedia Matrices v,7 and v.2 



Response Type 


Rank 


Comment 


Positive 






Matrix v. 1 


1 


Guides evaluator 




2 


Supports consistent feedback 




2 


Focuses attention 




4 


Simplifies application of principles/easy to use 


Matrix v.2 


i 


Simplifies application of principles/easy to use 




2 


Provides effective checklist format 


Negative/Change 






Matrix v. 1 


1 


Redundant 




1 


Inconsistent if used by many 




3 


Vocabulary too technical: define, simplify 




4 


Bad shape for response boxes 




4 


Add additional items 




4 


Reformat to checklist format 


Matrix v.2 


1 


Bad shape for response boxes 



Note. Rankings are based on items mentioned by more than 25% of the respondents. 
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Matrix for Guidelines for Designing Interactive Multimedia* 
Title of Multimedia Module 



Principle 1 . Related prior knowledge is the 
single most powerful influence in mediating 
subsequent learning . 


Principle 2. New knowledge becomes 
increasingly meaningful when integrated 
with existing knowledge. 


Application 1. Is information presented to 
accommodate ranges of prior knowledge of 
learners by: 


Application 2. Are structural aids to the new 
knowledge provided by: 

□ making the structure of the overall lesson 


□ layering and, abstracting, and/or providing 
various perspectives on new information? 


explicit? 

□ employing structural organizers, such as 


□ letting learners assemble their own connections 
with the concepts? 


headings to differentiate between critical and 
subordinate information? 

□ summaries of key relationships? 

□ elaboration strategies within the program or 
generated by the learner? 


Comments: 


Comments: 


Principle 3. Learning is influenced by the 
supplied organization of concepts to be 
learned. " ~ 


Principle 4. Knowledge to be learned needs 
to be organized in ways that reflect 
differences in learner familiarity with lesson 
content, the nature of the learning task and 
assumptions about the structure of 
knowledge. 


Application 3. Is information presented bv audio, 
video, and text: 


Application 4. Do links between/among nodes go 
beyond mere random access by providing 
movement: 


□ in consistent interface conventions, such as 
windows, links, menus for related ideas? 


□ from one concept to another? 


□ conceptually linked? 


□ from broad concept to specific data? 

□ with bottom up or top down hierarchies? 


Comments: 


Comments: 


Principle 5. Knowledge utility improves as 
processing and understanding deepen. 


Principle 6. Knowledge is best integrated 
when unfamiliar concepts can be related to 
familiar concepts. 


Application 5. Does the student have the 
opportunity to: 


Application 6. Are familiar visual, procedural, 
and/or verbal metaphors used for: 


□ reflect? 


□ lesson content? 


□ elaborate? 


□ system interface? 


Comments: 


Comments 



Principle 7. Learning improves as the 
number of complementary stimuli used to 
represent learning content increases . 


Principle 8 . Learning improves as the 
amount of invested mental effort increases. 


Application 7. Do sound, motion, text, and/or 
pictures present information in a way that: 


Application 8. Are learners required to expend 
mental effort through on screen elements that: 


□ relates directly to the information? 

□ complements the information? 


□ focus their attention by key information shown 
through diverse ways, such as highlighting? 




□ prompt them to predict, hypothesize, and/or 
generate new schema? 


Comments: 


Comments: 



Principle 9. Learning improves as 
competition for similar cognitive resources 
decreases and declines as competition for the 
same resources increases. 


Principle 10. Transfer improves when 
knowledge is situated in authentic contexts. 


Application 9. Is new and challenging 
information presented: 


Application 10. Is information presented in: 


□ using familiar multimedia conventions? 

□ without superfluous information? 


□ authentic, significant contexts? 


Comments: 


Comments: 


Principle 11. Knowledge flexibility 
increases as the number of perspectives on a 
given topic increases and the conditional 
nature of the knowledge is understood. 


Principle 12. Knowledge of details improves 
as instructional activities are more explicit, 
while understanding improves as the 
activities are more integrative. 


Application 11. Are learners helped to: 
□ access data in a variety of ways? 


Application 12. Is an orientation provided to the 
new information and its organization through 
the use of: 


□ work with data in multiple ways? 


□ pre-questions, objectives, advance organizers, 
and/or probability statements? 




□ questions posed about the information that is 
presented? 


Comments: 


Comments: 



Principle 13. Feedback increases the 
likelihood of learning response-relevant 
lesson content and decreases the likelihood 
of learning response-irrelevant lesson 
content. 


Principle 14. Shifts in attention improve 
the learning of related concepts . 


Application 13. Are the learners provided with 
opportunities for making: 


Application 14. Is kev data presented to gain 
attention through: 


□ ample responses about key concepts? 


□ design elements of highlighting, inverse 
display, change in color, fonts, and/or arrows? 


□ limited responses for incidentalfquestion- 
specific) information? 


□ procedural elements of a graphic overview 
(map of lesson structure) and/or prompts to 
relevant options? 

□ repetition throughout the module? 


Comments: 


Comments 


Principle 15. Learners become confused and 
disoriented when procedures are complex, 
insufficient, or inconsistent. 


Principle 16. Visual representations of 
lesson content and structure improve the 
learner y s awareness of both the conceptual 
relationships and the procedural requirements 
of a learning system. 


Application 15. Within the module are clear 
procedures given for 


Application 16. Are lesson content and structure 
interrelated by: 


□ navigating? 


□ concept maps? 


□ accessing support? 


□ graphical organizers? 


Comments: 


Comments: 



Principle 17. Individuals vary widely in 
their need for guidance. 


Principle 18. Learning systems are most 
efficient when they adapt to relevant 
individual differences . 


Application 17. Are learners helped In wavs to: 


Application 18. Does the module personalize the 
program: 


□ use the multimedia module itself, such as, how 
to access help or a glossary? 


□ at a nominal level by asking for learner's 
name, demographics, and/or preferences? 


□ use the multimedia module for learning, such 
as, feedback that is related to current status, or 
linked relationships ? 


□ at an advanced level by focusing on different 
rates of progress and providing relevant 
examples? 

□ at a conceptual level by adapting tasks to 
learners? 


Comments: 


Comments 



Principle 19. Metacognitive demands are 
greater for loosely structured learning 
environments than for highly structured 
ones . 


Principle 20. Learning is facilitated when 
system features are functionally self-evident, 
logically organized, easily accessible, and 
readily displayed. 


Application 19. Can the learner monitor progress 
and/or learning strategies by: 

□ accessing prompts? 

□ using self-checks? 

□ creating linkages? 

□ asking questions? 

□ reviewing? 


Application 20. Are on-screen elements designed 
to: 

□ simplify learner's use of the module? 

□ help learners assess their progress in learning 
the concepts? 


Comments: 


Comments: 


Total number oi 
C 

Overall a 


checked items: 

1 

3mment s: 



*The Matrix is based on: Park, I. & Hannafin, M. J. (1993). Empirically-based guidelines for the design 
of interactive multimedia. Educational Technology Research and Development. 43. 3, 63-85. The 20 
Principles cited in this document are quoted directly from Park and Hannafin. ETR&D is published by 
the Association for Educational Communications and Technology the copyright holder. The" 
Principles" are reprinted in the Matrix with permission of the publisher. 

Matrix completed 

by 



Matrix designed by Annette Sherry, Dept, of Educational Technology, College of Education, University 
of Hawaii at Manoa, 1996 




391 



12 




U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 

REPRODUCTION BASIS 




This document is covered by a signed “Reproduction Release 
(Blanket)” form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket’^. . 



ERiC 



